Suboptimality of constrained least squares and improvements via non-linear predictors

We study the problem of predicting as well best linear predictor in a bounded Euclidean ball with respect to squared loss. When only boundedness data generating distribution is assumed, we establish that least squares estimator constrained does not attain classical O(d?n) excess risk rate, where d dimension covariates and n number samples. In particular, construct such incurs an order ?(d3?2?n) hence refuting recent conjecture Ohad Shamir [JMLR 2015]. contrast, observe non-linear predictors can achieve optimal rate no assumptions on covariates. discuss additional distributional sufficient guarantee for estimator. Among them are certain moment equivalence often used robust statistics literature. While central analysis unbounded heavy-tailed settings, our work indicates some cases, they also rule out unfavorable distributions.